Skip to content

refactor: stateful Solver instances and two-step solve API#682

Open
FabianHofmann wants to merge 28 commits into
masterfrom
solver-refac
Open

refactor: stateful Solver instances and two-step solve API#682
FabianHofmann wants to merge 28 commits into
masterfrom
solver-refac

Conversation

@FabianHofmann
Copy link
Copy Markdown
Collaborator

@FabianHofmann FabianHofmann commented May 13, 2026

closes #628 #583

Changes proposed in this Pull Request

Refactor of the solver layer to put solver state on a stateful Solver instance and expose a clean construct-then-solve workflow.

  • Stateful Solver on Model.solver. Solver state (native model, results) now lives on a Solver instance attached to Model.solver. Model.solver_model and Model.solver_name become read-only properties delegating to model.solver (assigning anything but None raises; setting None closes the solver). Model.solver_name may be None before a solve. These two properties are candidates for future deprecation.
  • Construct-then-solve API. Build a solver via Solver.from_name(name, model, io_api=..., options=...) (or SolverClass.from_model(model, ...)), then call solver.solve() to run and obtain a Result, and model.apply_result(result) to write the solution back to the model. Solver is a dataclass; subclasses no longer need __init__ overrides.
  • Clear lifecycle hooks per solver. Each subclass overrides at most three methods: _build_direct(**kwargs) (build the native model from self.model), _run_direct(**kwargs) (run the prebuilt native model), and _run_file(**kwargs) (invoke the solver on self._problem_fn). File-only solvers (CBC, GLPK, Cplex, SCIP, Xpress, Knitro, COPT, MindOpt) override only _run_file. Direct-API solvers (Highs, Gurobi, Mosek, cuPDLPx) override all three.
  • Legacy entry points deprecated. Solver.solve_problem, Solver.solve_problem_from_model, and Solver.solve_problem_from_file are kept on the base class as thin shims that route through the new pipeline and emit DeprecationWarning. To be removed in a future release.
  • Two ways to get the native solver model. Either via the module-level / Model-bound helpers (model.to_gurobipy(), model.to_highspy(), to_cupdlpx(model)), or directly via Solver.from_model(model, io_api="direct").solver_model. The previous public Solver.to_solver_model method is removed and folded into the internal _build_direct hook to avoid exposing a third redundant path.
  • Declarative solver capabilities. Capabilities are declared as features: frozenset[SolverFeature] ClassVars on each Solver subclass; query with Solver.supports(feature). SolverFeature is exported from linopy (and linopy.solvers); linopy.solver_capabilities remains as a back-compat shim with a lazy SOLVER_REGISTRY mapping.
  • Result carries solver report. Result gains solver_name and report: SolverReport | None (runtime, MIP gap, dual bound, iteration counts) and prints them in __repr__. CBC, HiGHS, Gurobi, Knitro, and cuPDLPx populate report; when possible also populate the MIP dual_bound.
  • Label-indexed solution mapping. Solution.primal and Solution.dual are now dense np.ndarrays indexed by linopy label (length = max_label + 1); masked or solver-dropped slots are NaN. Previously pd.Series keyed by name. Each solver emits arrays in this label-indexed form — direct-API solvers via cached _vlabels/_clabels populated at _build_direct time, file-based solvers via the shared _solution_from_names helper. Robust to solver iteration order and to solvers dropping unused variables (e.g. CPLEX on LP read).
  • Cached constraint labels. Constraints exposes a cached label_index (mirroring Variables.label_index), so apply_result and IIS extraction no longer trigger a full model.matrices rebuild. Invalidated on add/remove.

Reviewer requests

  • @FBumann: align SolverReport with SolverMetrics (feat: add unified SolverMetrics #583); expose dual_bound (best bound)
  • @coroa: make Solver a dataclass; drop the solver_class(**solver_options) pattern
  • @coroa: add Solver.from_name(name, model, io_api=..., options=...) static constructor + SolverClass.from_model(...) classmethod
  • @coroa: drop prepare_solver / run_solver two-step API
  • @coroa: cache vlabels / clabels on the solver instead of going through model.matrices, see vlabels-clabels-flow.html for more details
  • @coroa: adopt the Model.solve = self.solver = Solver.from_name(...); apply_result(...) pattern in Model.solve
  • @coroa: add is_available() classmethod per solver + lazy SOLVER_REGISTRY so available_solvers discovery doesn't grab licenses prematurely
  • @coroa: refactor OETC as a Solver subclass
  • @coroa: keep the interface extensible for asynchronous solving (Gurobi batch optimization, OETC) — return early with a job handle and retrieve later
  • @coroa: incremental update path — solver holds a shallow copy of the linopy model, solver.update(model) diffs and pushes only changed bounds/rhs/coefficients (CoW on variable/constraint data)

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

FabianHofmann and others added 16 commits May 12, 2026 06:24
Phase B of solver refactor (issue #628). Makes the Solver instance the
canonical owner of solver-side state.

- Base Solver.__init__ now initializes options, status, solution, report,
  solver_model, io_api, env, capability, _env_stack.
- Adds to_solver_model / update_solver_model / resolve / close / __del__
  on the base class; resolve dispatches to per-subclass _resolve.
- Adds _make_result helper that populates instance state and stamps
  solver_name and report onto Result.
- Gurobi: env creation moved off per-call ExitStack onto self._env_stack
  so the env remains valid after solve returns; to_solver_model and
  _resolve overrides wired.
- Highs / Mosek / cuPDLPx: to_solver_model + _resolve overrides; Mosek
  task is now kept alive via self._env_stack instead of being closed at
  function exit.
- CBC / GLPK / Cplex / SCIP / Xpress / Knitro / COPT / MindOpt: minimal
  wiring — populate self.status/self.solution/self.solver_model/self.io_api
  via _make_result and pass solver_name + report (where readily
  available) into the returned Result.

solve_problem dispatcher and the public solve_problem_from_model /
solve_problem_from_file signatures are unchanged. Model.solve is
untouched (Phase C).
Surfaces solver name, status, io_api, and solution/report summary.
Move SolverFeature and _xpress_supports_gpu into linopy.solvers; declare
features/display_name as ClassVars on each Solver subclass with a
Solver.supports() classmethod. solver_capabilities becomes a back-compat
shim with a lazy SOLVER_REGISTRY mapping. Model.solve uses the class API
directly; SolverFeature is re-exported at the package top level.
Stash `sense` on the Solver instance in `to_solver_model` and make
`Solver.resolve()` take no args. Add `Model.to_solver_model(name)` and
`Model.resolve()` wrappers so the two-step direct-API flow lives on the
model. Update the direct-API test and re-run the piecewise notebook.
Model.to_solver_model -> prepare_solver and Model.resolve -> run_solver
(plus Solver.resolve/_resolve -> run/_run). Avoids the awkward "resolve
on first call" reading. Solver.to_solver_model is kept since it
accurately produces the native solver model.
Replace Xpress-specific _xpress_supports_gpu with a generic
_installed_version_in helper, and add Solver.runtime_features() as an
override hook for version/env-conditional capabilities. Xpress now
declares its GPU support via runtime_features() instead of inline
frozenset arithmetic on the class body.
…method

Move the full parameter docstring onto Solver.solve_problem_from_model and
drop the per-subclass duplicates on Mosek and cuPDLPx; subclasses now inherit
the abstract method's docstring.
Unify per-solver _translate_to_* methods under a common _build_solver_model
name, hoist their local imports to module top-level, drop dead params from
cuPDLPx (moving its UserWarning into the public to_solver_model), and add
TYPE_CHECKING stubs. Expand to_* deprecation messages with step-by-step
migration paths, wrap existing tests in pytest.warns, and cover the
unknown-solver-name branch in prepare_solver.
@FBumann
Copy link
Copy Markdown
Collaborator

FBumann commented May 13, 2026

@FabianHofmann I suggest checking the new class SolverReport against my SolverMetrics in #583, the purpose is pretty much the same. This will also probably close #583.
One thing we definitely need is SolverReport.dual_bound (also known as best bound).
Peak memory was also added in #583, but might be less commonly exposed by the solver. I also dont really need it...

@coroa
Copy link
Copy Markdown
Member

coroa commented May 13, 2026

Can we make the solver class into a data class, too? And get rid of this strange instantiate solver_class(**solver_options) pattern.

I am not sure whether there is a benefit to holding a solver class instance without a model attached, so what about constructors like:

Solver.from_name("gurobi", linopy_model, io_api=..., options=options)
# a staticmethod

which dispatches to

Gurobi.from_model(linopy_model, io_api=..., options=options)
# the specific classmethod, even though that is also implemented on the main Solver class

which dispatches according to io_api.

I'd say this then gets rid of any case for prepare_solver or some such.

@coroa
Copy link
Copy Markdown
Member

coroa commented May 13, 2026

Benchmark: OETC should be able to become a solver class.

class Model:
    ...
    def solve(solver_name, io_api, **solver_options):
        self.solver = None
        self.solver = solver = Solver.from_name(solver_name, model=self, io_api=io_api, options=solver_options)
        result = solver.solve()
        self.apply_result(result)

@coroa
Copy link
Copy Markdown
Member

coroa commented May 13, 2026

Solvers should get a class method:

class Gurobi:
    @classmethod
    def is_available():
        try:
            from gurobipy import Model
            with Model():
                  return True
         except ImportError, LicenseError?:
             return False

We then use a derived Collection pattern similar to the following for available_solvers so not to grab licenses prematurely.

https://github.com/snakemake/snakemake/pull/3900/changes#diff-857b2ff1aff916efdddad6bcd645a90f388276460cf75e6c9a0362745a70371fR13-R58

@coroa
Copy link
Copy Markdown
Member

coroa commented May 13, 2026

ah yes, and OETC and some Gurobi compute like instances can have an asynchronous solving option. Where you do not want to block the process, but return early and only give back some sort of job identifier in hand with which to retrieve the solution later.

Gurobi docs: https://docs.gurobi.com/projects/optimizer/en/current/features/batchoptimization.html

And i don't mean implement this now, here, but the interface should be extensible to allow for that.

@coroa
Copy link
Copy Markdown
Member

coroa commented May 14, 2026

i am unsure how to effectively keep track of the state that was communicated to the solver. ie. when you then do modifications on the linopy model data after it was communicated to the solver. when do you send those updates (and which updates)

the most promising way in my mind would be the following:

the solver object holds a shallow copy of the linopy model (up until the individual constraint and variable objects). EDIT: probably only of the constraints and variable objects.

when you make an update to a variable bound or constraint you use some sort of cow to create a copy in the linopy model (this mostly means something like a v.data = v.data.assign pattern and maybe on mutable constraints and rhs an explicit cow numpy flag).

this means you share variable data/constraint data until you make the first modification.

then you do solver.update(linopy_model) which does diffing and sends changes to solver.

several nice characteristics that way:

  1. you don't send changes prematurely (ie. immediately communicate any change on rhs assignment)
  2. sending changes is minimal/ incremental
  3. its compatible with putting solver connections into a separate thread
  4. update does not need to care whether it is effectively the same model or a completely newly constructed one
  5. memory use is efficient

@coroa
Copy link
Copy Markdown
Member

coroa commented May 14, 2026

@FabianHofmann i reread the pr description. don't use vlabels, clabels through m.matrices. m.matrices is expensive when constraints are not frozen. but I think you know and that's why they are stored on solver, isn't it

@FabianHofmann
Copy link
Copy Markdown
Collaborator Author

@FabianHofmann I suggest checking the new class SolverReport against my SolverMetrics in #583, the purpose is pretty much the same. This will also probably close #583. One thing we definitely need is SolverReport.dual_bound (also known as best bound). Peak memory was also added in #583, but might be less commonly exposed by the solver. I also dont really need it...

thanks @FBumann for raising this. I pulled over the dual_bound feature from your pr. should cover all if it then

Replace pd.Series with dense NaN-padded np.ndarray keyed by integer
model labels. Adds values_to_lookup_array helper; simplifies
Model.apply_result; migrates every solver to build the lookup array
directly (direct-API solvers via cached _vlabels/_clabels, file-based
solvers via a shared _names_to_labels helper). Drops the now-unused
series_to_lookup_array and all name-based solution fallbacks.
Adds dual_bound field to SolverReport, populated by HiGHS, Gurobi
and Knitro. Declared via new SolverFeature.MIP_DUAL_BOUND_REPORT
so tests gate assertions on capability instead of solver names.
pre-commit-ci Bot and others added 5 commits May 15, 2026 06:48
Mirror VariableLabelIndex on the constraint side: add
ConstraintBase.active_labels (cheap on both Constraint and CSRConstraint)
and a cached ConstraintLabelIndex on Constraints. Model.apply_result and
IIS extraction now read vlabels/clabels from these caches instead of
self.matrices, which previously rebuilt the full A matrix on every solve.
…rder

Restore the 43a82aa contract: Solution.primal and Solution.dual are
dense ndarrays indexed by linopy label with NaN at gaps. Each solver
builds the label-indexed form itself; apply_result shrinks to a direct
lookup_vals.

This fixes two classes of bugs the intermediate 'primal/dual in build
order' contract couldn't handle:

- File-based LP solvers iterate variables in objective-encounter order,
  which can interleave entries from multiple add_variables calls (e.g.
  non-aligned coords producing x0, x10, x1, x11, ...). Positional
  alignment with vlabels broke for every test exercising masked or
  non-aligned models on CBC / GLPK / Cplex / SCIP / Xpress.
- CPLEX drops entirely-unconstrained unused variables on LP read, so
  get_values() returns fewer values than the linopy model has labels;
  positional alignment errored at apply_result.

Implementation:

- _solution_from_names(values, names): file-based path. Parses linopy
  labels via the existing _names_to_labels helper and scatters values
  into a label-indexed array. Used by CBC, GLPK, Cplex, SCIP, Xpress,
  Knitro, COPT, MindOpt, and the from_file branches of Highs/Gurobi.
- _solution_from_labels(values, labels): direct-API path. Uses cached
  vlabels/clabels populated on the Solver instance at to_solver_model
  time. Used by Highs, Gurobi, Mosek, cuPDLPx.
- Highs and Gurobi from_file branches converge on _solution_from_names,
  replacing the inline keep + argsort pattern.
Solver is now a dataclass; subclasses inherit init. Construction routes
through Solver.from_name(name, model, io_api=..., options=...) which
builds at construction (direct API or LP/MPS file). solver.solve()
returns a Result; model.apply_result(result) writes the solution back.
Drops Model.prepare_solver/run_solver; to_* io helpers no longer warn.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Solver Refactor and Extension

3 participants